Conventional light microscopy is one of the most widely used and efficient techniques of examining and diagnosing biological samples. However, said techniques are limited by several factors. Stain colors may change or fade over time. This applies particularly to immunofluorescence staining techniques, which are frequently usable during a very short period of time only [14]. Biological samples tend to age. However, slides that are used are unique and cannot be reproduced, e.g. when the slide glass breaks. Unfortunately, microscopes visualize only a small part of a slide at any one time, and the magnifications are limited to the available objectives. Simultaneous viewing of a slide is limited to a very small number of persons, and the slide has to be physically present. In addition, it is not possible to annotate regions of interest on the slide.
One goal of virtual microscopy consists in solving said problems by replacing direct work with the microscope and the slide with utilization of digitized samples. To this end, the slide is scanned once by a fully automated robotic microscope and is digitally stored. Such digitized samples exhibit none of the above-mentioned problems. In addition, it is possible to examine such digitized samples from a distance from any computer as if the sample was physically present. Any regions of interest, for example cancerous cells, can be annotated on the slide, and consequently, it is still possible to understand at a later point in time why a certain examining person made a specific diagnosis. Consequently, in the long term, virtual microscopy may possibly replace traditional microscopy.
However, there are still problems that have not been entirely solved. One of said problems consists in the need to correctly register (align) any fields of view in biological samples with larger areas of no information content, such as samples from cytometry. A slide is simply too large to be detected with one image at a reasonable magnification. One may imagine, for example, the case of a 20-fold magnification and of a typical color camera producing a 1000×1000 pixel image and comprising a cell size of 7.4 μm×7.4 μm. This would mean that each pixel covered an area of 0.37 μm×0.37 μm, and that each field of view had an area of 0.37 mm×0.37 mm. Of course, this depends on the type of sample being scanned, but to simply give an example, the valid area to be scanned may be 2 cm×3 cm, for example, which would mean that about 55×82=4510 fields of view would have to be detected. The scanning operation of the slide to be virtualized is typically performed by computer-controlled, motorized positioning tables (stages) which, additionally, may be accurately calibrated. However, irrespective of the degree of precision of their calibration, positioning tables will invariably have positioning errors which may accumulate, in turn, during the scanning operation and may consequently result in alignment errors in the virtual image of the slide. As a result, a great challenge in virtual microscopy is to develop algorithms which mutually align any detected fields of view in such a manner that the originally scanned slide is fully reconstructed.
The conventional approach to stitching fields of view in the manner of a mosaic (mosaicing) consists in scanning the slide in a first step and, in a second step, to align each field of view with its neighboring fields of view, which is sometimes followed by an optimization step. In a naïve approach, the fields of view are simply mutually aligned one after the other in the sequence in which they have been detected. It is better to take into account the mosaic by optimization while considering the interaction effects between the change in the relative locations of two fields of view on the matching of the other field-of-view pictures.
In other words, in virtual microscopy there is the problem of joining (stitching) adjoining microscopic views to form a contiguous larger image also referred to as a mosaic. A microscope slide is digitized in that a computer-controlled stage is displaced in a controlled manner, a picture being taken following each displacing operation. The stage is displaced by a defined travel in each case, so that the next field of view either directly borders on the preceding one or overlaps the preceding one in a defined manner. However, the mechanics of microscope stages is inaccurate, so that there will be mispositionings. Therefore, the captured fields of view are typically registered toward one another on the part of software in a second step. In this context, different similarity measures such as area-based SSD or cross-correlation or least squares correlation as well as feature-based measures are employed. The precise positioning offset, or the transform between two images, is determined by determining the position in which the similarity measure reaches its maximum in the overlap area.
The transforms thus determined may be used for stitching the mosaic in a sequential manner, such as in the shape of a meander. However, since registration determination on the part of software is also faulty, such an approach will see an accumulation of alignment errors, and there will be a visual offset of the fields of view in the mosaic.
The need for a comprehensively operating optimization scheme becomes obvious when considering three images which have been aligned against one another and thus result in three transforms such as translational displacements about an offset vector in order to mutually align the three images correctly. In a perfect environment, said transforms will result in an overdetermined but consistent equation system: when concatenated they will result in an identity transform. However, since the transforms are measurements on real-world images exposed to noise and other deteriorations, the transforms will be susceptible to errors. The idea of optimization consists in minimizing the average measurement errors, for example by equally distributing same over all of the transforms.
There are several approaches which try to solve the problem that misalignments may accumulate in the event of a sequential alignment of the subimages. As was described above, the transforms should not only be taken into account sequentially, but the transforms from all of the overlap areas should be taken into account, such as in [7], for example. In this context, an equation system is set up which may be solved by means of an equalization calculation or other optimization methods. The technique described in [6] also falls into the category of said approaches.
Since the transforms determined are based on observations, not all of them are exact. Any solution of an equation system thus disturbed will therefore still be erroneous. An approach which minimizes said errors would be desirable.
Some works have already comprehensively addressed the problem set forth above. Said mosaicing algorithms may typically be categorized into two separate steps. In the first step, neighboring images are aligned in a pairwise manner by applying an area-based or feature-based registration algorithm. In a second step, the transform parameters thus estimated are then bundle-adjusted to obtain a globally consistent transform space. Szeliski [3] has presented an authoritative survey of both local and global image stitching methods. However, Szeliski's focus is on aligning multiple viewpoint images and producing panoramic images under affine transform constraints. However, this poses a more general problem than arises in virtual microscopy, where the transform space of the images is strictly translational. Consequently, in virtual microscopy, more specialized mosaicing algorithms may be used in order to obtain an optimum mosaic.
Davis [4] addresses the stitching of scenes with moving objects. Davis obtained a system of linear equations from pairwise alignments and solved the system by using a least squares approach to obtain a globally optimized transform space. What is problematic about Davis' approach is that by solving the system in a least error squares sense, Davis distributes the error equally over all transforms. However, this is justified only if the error in the transforms is strictly Gaussian in shape, which is not always the case. Kang et al. [5] proposes a graph-theoretic approach for global registration of 2D mosaics under projective secondary transform constraints. Surprisingly, there are only few articles directly dealing with stitching virtual slides. Sun et al. use feature matching with the Harris Corner detector and perform global geometric correction with an objective function which minimizes the Euclidean distance between the feature points when the transform is applied [6]. Appleton et al. address the image stitching problem as a global optimization problem [7]. They use dynamic programming and a similarity function in order to place a complete row of images with previously placed rows at a point in time. However, since they place only one row at a time, this is only partly a global optimization, which, in order to be complete, would involve placing all of the images at once. In addition, the process presented there would use an overlap of 45% between the images in order to achieve good results. This, again, results in a very long scan time. In [8], Davis' [4] idea of producing a linear equation system was used for the optimization problem and was weighted by weighting each transform according to its reliability. Consequently, the equation system was solved in a weighted least error squares sense. This approach entailed objectively improved results as compared to the unweighted approach. However, under specific conditions, even the latter approach still exhibits errors. It is mostly with slides having significantly large areas of low information content and, consequently, low correlation values that some of the subimages in these areas are not correctly repositioned.
In [14], a method of automatically creating a virtual microscope slide is described. Images are positioned exclusively on the basis of the erroneous positioning properties of the stage.
In [18], a method of software-related registration of virtual microscope slides is described. However, what is registered is only ever that part of the microscope slide which is currently rendered on the screen. Mosaic registration is effected sequentially.
In [17], a system of digitizing slides is described, but no registration algorithmics.
In [15], a method of automatically creating a virtual microscope slide is described. The system creates a microscope slide in that each image is automatically registered with the preceding one. A correction value is calculated from the offset information in terms of control engineering, and the stage is displaced in a corrected manner for capturing the next image. By using this method, a relatively precise virtual microscope slide may be created even by means of relatively imprecise stages. The accuracy of the stage is therefore taken into account, specifically in order to correct the stage in terms of control engineering. Mosaic registration is effected sequentially.
It would be desirable to have a concept for generating a mosaic picture of an object plane which functions in a more stable and, thus, in a qualitatively improved manner.